Method and apparatus for compressing neural network

ABSTRACT

A method and apparatus for compressing a neural network are provided. A specific embodiment of the method includes: acquiring a to-be-compressed trained neural network; selecting at least one layer from layers of the neural network as a to-be-compressed layer; performing following processing steps sequentially on each of the to-be-compressed layers in descending order of the number of level of the to-be-compressed layer: determining a pruning ratio based on a total number of parameters included in the to-be-compressed layer, selecting a parameter for pruning from the parameters included in the to-be-compressed layer based on the pruning ratio and a parameter value threshold, and training the pruned neural network based on a preset training sample using a machine learning method; and determining the neural network obtained after performing the processing steps on the selected at least one to-be-compressed layer as a compressed neural network, and storing the compressed neural network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese Patent Application no.201711473963.3, filed with the China National Intellectual PropertyAdministration (CNIPA) on Dec. 29, 2017, the contents of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

Embodiments of the present disclosure relate to the field of computertechnology, specifically to the field of Internet technology, and morespecifically to a method and apparatus for compressing a neural network.

BACKGROUND

Recently, with the continuous development of artificial intelligence,the application range of neural networks is also constantly expanding.Here, the neural network is an abbreviation of artificial neuralnetwork. The neural network may be applied to a server to processimages, texts, audios, and the like. Certainly, now the neural networkmay also be included in a client application. A user may edit images,texts, audios, etc. through the neural network in the client applicationinstalled on the terminal device.

Existing neural networks usually take up a lot of storage space, such asdisk space or memory space. If the users install a lot of applicationsincluding neural networks on their device (e.g., a mobile device such asa smartphone or a tablet), the device may have less available storagespace, and abnormal conditions such as slow running and downtime mayoccur on the device.

SUMMARY

Embodiments of the present disclosure propose a method and apparatus forcompressing a neural network.

In a first aspect, the embodiments of the present disclosure provide amethod for compressing a neural network, including: acquiring ato-be-compressed trained neural network; selecting at least one layerfrom layers of the neural network as a to-be-compressed layer;performing the following processing steps sequentially on each of theto-be-compressed layers in descending order of the number of level ofthe to-be-compressed layer in the neural network: determining a pruningratio based on a total number of parameters included in theto-be-compressed layer, selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and a parameter value threshold, and training the pruned neuralnetwork based on a preset training sample using a machine learningmethod; and determining the neural network obtained after performing theprocessing steps on the selected at least one to-be-compressed layer asa compressed neural network, and storing the compressed neural network.

In some embodiments, the selecting at least one layer from layers of theneural network as a to-be-compressed layer includes: selecting, inresponse to the neural network including a convolutional layer and afully connected layers, at least one of at least one convolutional layeror at least one fully connected layers as the to-be-compressed layer.

In some embodiments, the selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and a parameter value threshold, and training the pruned neuralnetwork based on a preset training sample using a machine learningmethod includes: performing the following pruning training operations:determining a product of a specified ratio and the total number as apruning number, and selecting, from the parameters included in theto-be-compressed layer, the pruning number of parameters having anabsolute value of a parameter value not greater than the parameter valuethreshold for pruning in ascending order of the absolute value of theparameter value, wherein the specified ratio is not greater than thepruning ratio; training the pruned neural network based on the trainingsample using the machine learning method; and stopping execution of thepruning training operations in response to determining that an accuracyof the current trained neural network is lower than a preset accuracy,or a ratio of the number of the pruned parameters to the total number inthe to-be-compressed layer is not lower than the pruning ratio; andresetting the specified ratio based on the pruning ratio and the ratio,and continuing the pruning training operations in response todetermining that the accuracy of the current trained neural network isnot lower than the preset accuracy and the ratio is lower than thepruning ratio.

In some embodiments, the resetting the specified ratio based on thepruning ratio and the ratio includes: subtracting the ratio from thepruning ratio to obtain a first ratio, and resetting the specified ratioto a positive number not greater than the first ratio.

In some embodiments, the resetting the specified ratio to a positivenumber not greater than the first ratio includes: resetting thespecified ratio to a positive number not greater than the first ratioand a current value of the specified ratio.

In some embodiments, the resetting the specified ratio to a positivenumber not greater than the first ratio and a current value of thespecified ratio includes: resetting the specified ratio to a positivenumber less than the current value and not greater than the first ratio.

In a second aspect, the embodiments of the present disclosure provide anapparatus for compressing a neural network, including: an acquisitionunit, configured to acquire a to-be-compressed trained neural network; aselection unit, configured to select at least one layer from layers ofthe neural network as a to-be-compressed layer; a processing unit,configured to perform following processing steps sequentially on each ofthe to-be-compressed layers in descending order of the number of levelof the to-be-compressed layer in the neural network: determining apruning ratio based on a total number of parameters included in theto-be-compressed layer, selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and a parameter value threshold, and training the pruned neuralnetwork based on a preset training sample using a machine learningmethod; and a storing unit, configured to determine the neural networkobtained after performing the processing steps on the selected at leastone to-be-compressed layer as a compressed neural network, and store thecompressed neural network.

In some embodiments, the selection unit includes: a selection subunit,configured to select, in response to the neural network including aconvolutional layer and a fully connected layers, at least one of atleast one convolutional layer or at least one fully connected layers asthe to-be-compressed layer.

In some embodiments, the processing unit includes: a first processingsubunit, configured to perform the following pruning trainingoperations: determining a product of a specified ratio and the totalnumber as a pruning number, and selecting, from the parameters includedin the to-be-compressed layer, the pruning number of parameters havingan absolute value of a parameter value not greater than the parametervalue threshold for pruning in ascending order of the absolute value ofthe parameter value, wherein the specified ratio is not greater than thepruning ratio; training the pruned neural network based on the trainingsample using the machine learning method; and stopping execution of thepruning training operations in response to determining that an accuracyof the current trained neural network is lower than a preset accuracy,or a ratio of the number of the pruned parameters to the total number inthe to-be-compressed layer is not lower than the pruning ratio; and asecond processing subunit, configured to reset the specified ratio basedon the pruning ratio and the ratio, and continue the pruning trainingoperations in response to determining that the accuracy of the currenttrained neural network is not lower than the preset accuracy and theratio is lower than the pruning ratio.

In some embodiments, the second processing subunit includes: a settingmodule, configured to subtract the ratio from the pruning ratio toobtain a first ratio, and reset the specified ratio to a positive numbernot greater than the first ratio.

In some embodiments, the setting module includes: a setting submodule,configured to reset the specified ratio to a positive number not greaterthan the first ratio and a current value of the specified ratio.

In some embodiments, the setting submodule is further configured to:reset the specified ratio to a positive number less than the currentvalue and not greater than the first ratio.

In a third aspect, the embodiments of the present disclosure provide anelectronic device, including: one or more processors; and a storageapparatus, for storing one or more programs, the one or more programs,when executed by the one or more processors, cause the one or moreprocessors to implement the method according to any one of theembodiments in the first aspect.

In a fourth aspect, the embodiments of the present disclosure provide anon-transitory computer readable storage medium, storing a computerprogram thereon, the program, when executed by a processor, implementsthe method according to any one of the embodiments in the first aspect.

The method and apparatus for compressing a neural network provided bythe embodiments of the present disclosure select at least one layer fromlayers of the acquired to-be-compressed trained neural network as ato-be-compressed layer, perform specified processing steps sequentiallyon each of the to-be-compressed layers in descending order of the numberof level of the to-be-compressed layer in the neural network todetermine the neural network obtained after performing the processingsteps on the selected at least one to-be-compressed layer as acompressed neural network and store the compressed neural network.Therefore, the specified processing steps performed on the selectedto-be-compressed layer are effectively utilized to appropriately prunethe parameters in the to-be-compressed layer, and the pruned neuralnetwork based on a preset training sample is trained using the machinelearning method, so that the neural network may be restored to theoriginal accuracy as much as possible, thereby achieving effectivecompression of the neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

After reading detailed descriptions of non-limiting embodiments withreference to the following accompanying drawings, other features,objectives and advantages of the present disclosure will become moreapparent:

FIG. 1 is an exemplary system architecture diagram to which the presentdisclosure may be applied;

FIG. 2 is a flowchart of an embodiment of a method for compressing aneural network according to the present disclosure;

FIG. 3 is a schematic diagram of an application scenario of the methodfor compressing a neural network according to the present disclosure;

FIG. 4 is a schematic structural diagram of an embodiment of anapparatus for compressing a neural network according to the presentdisclosure; and

FIG. 5 is a schematic structural diagram of a computer system adapted toimplement an electronic device of the embodiments of the presentdisclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

The present application will be further described below in detail incombination with the accompanying drawings and the embodiments. Itshould be appreciated that the specific embodiments described herein aremerely used for explaining the relevant disclosure, rather than limitingthe disclosure. In addition, it should be noted that, for the ease ofdescription, only the parts related to the relevant disclosure are shownin the accompanying drawings.

It should also be noted that the embodiments in the present applicationand the features in the embodiments may be combined with each other on anon-conflict basis. The present application will be described below indetail with reference to the accompanying drawings and in combinationwith the embodiments.

FIG. 1 shows an illustrative architecture of a system 100 which may beused by a method for compressing a neural network or an apparatus forcompressing a neural network according to the embodiments of the presentapplication.

As shown in FIG. 1, the system architecture 100 may include servers 101and 103, and a network 102. The network 102 serves as a medium providinga communication link between the servers 101 and 103. The network 102may include various types of connections, such as wired or wirelesstransmission links, or optical fibers.

The server 101 may be a server providing various services, for example,a data storage server for storing the trained neural network.

The server 103 may be a server providing various services, for example,a server for compressing a neural network. The server may acquire ato-be-compressed trained neural network, analyze the neural network, andstore a processing result (for example, a compressed neural network).

It should be noted that the method for compressing a neural networkaccording to the embodiments of the present application is generallyexecuted by the server 103. Accordingly, the apparatus for compressing aneural network is generally installed on the server 103.

It should be noted that if the neural network acquired by the server 103is prestored locally, the system architecture 100 may do not include theserver 101.

It should be appreciated that the numbers of the servers and thenetworks in FIG. 1 are merely illustrative. Any number of servers andthe networks may be provided based on the actual requirements.

With further reference to FIG. 2, a flow 200 of an embodiment of themethod for compressing a neural network according to the presentdisclosure is illustrated. The flow 200 of the method for compressing aneural network includes the following steps:

Step 201, acquiring a to-be-compressed trained neural network.

In the present embodiment, the electronic device (e.g., the server 103as shown in FIG. 1) on which the method for compressing a neural networkoperate may acquire a to-be-compressed trained neural network from aserver in remote communication connection (e.g., the server 101 as shownin FIG. 1) through a wired connection or a wireless connection.Certainly, if the neural network is pre-stored locally in the electronicdevice, the electronic device may also acquire the neural networklocally.

It should be noted that the above neural network may be a neural networkoccupying space exceeding an occupancy threshold. Further, the neuralnetwork may be a neural network that occupies space exceeding theoccupancy threshold and is included in a client application, such as aclient application suitable for mobile devices. When the neural networkin the client application occupies large storage space, by compressingthe neural network, the disk space or memory space of the terminaldevice on which the client application is installed may be saved.Moreover, when the user downloads the client application, the user'swaiting time may be reduced, and the consumption of traffic may bereduced.

In addition, the neural network acquired by the electronic device mayinclude, for example, at least one input layer, at least one hiddenlayer, and at least one output layer. Here, each layer of the neuralnetwork may have a corresponding number of level. For example, assumingthat the neural network includes one input layer, one hidden layer, andone output layer, the input layer may be in the first layer of theneural network, and the number of level of the input layer may be 1; thehidden layer may be in the second layer of the neural network, and thenumber of level of the hidden layer may be 2; and the output layer maybe in the third layer of the neural network, and the number of level ofthe output layer may be 3.

It should be noted that the neural network may refer to the artificialneural network (ANN). A neural network is usually an operational modelconsisting of a large number of nodes (or neurons) connected to eachother. Each node may represent a specific output function, which isreferred to as an activation function. The connection between every twonodes represents a weighted value for passing the connection signal,which is referred to as a weight and is deemed as the memory of theartificial neural network. Common neural networks include, for example,deep neural network (CNN), convolutional neural network (CNN), andrecurrent neural network (RNN).

Step 202, selecting at least one layer from layers of the neural networkas a to-be-compressed layer.

In the present embodiment, the electronic device may select at least onelayer from layers of the neural network as a to-be-compressed layer. Forexample, the electronic device may select each layer of the neuralnetwork as the to-be-compressed layer.

In some alternative implementations of the present embodiment, inresponse to the neural network including a convolutional layer and afully connected layers (PC), the electronic device may select at leastone convolutional layer and at least one fully connected layers as theto-be-compressed layers.

Step 203, performing processing steps sequentially on each of theto-be-compressed layers in descending order of the number of level ofthe to-be-compressed layer in the neural network.

In the present embodiment, after selecting the to-be-compressed layerfrom the acquired neural network, the electronic device may perform thefollowing processing steps sequentially on each of the to-be-compressedlayers in descending order of the number of level of theto-be-compressed layer in the neural network: determining a pruningratio based on the total number of parameters (weights) included in theto-be-compressed layer, selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and a parameter value threshold (e.g., 0.1), and training thepruned neural network based on a preset training sample using a machinelearning method. It should be noted that the parameter value thresholdmay be adjusted according to actual needs, and the present, embodimentdoes not have any limitations with this respect. In addition, theelectronic device may implement pruning of the parameter by setting theselected parameter to a specified value (e.g., 0).

For example, the electronic device may be pre-stored with a set of valueranges locally, where each of the value ranges may correspond to apruning ratio. The electronic device may first find a value range in theset of value ranges in which the total number of parameters included inthe to-be-compressed layer is located, and determine the pruning ratiocorresponding to the value range as the pruning ratio corresponding tothe to-be-compressed layer. Then, the electronic device may calculatethe product of the pruning ratio and the total number, and determine theproduct as the pruning number. Then, the electronic device may select,from the parameters included in the to-be-compressed layer, the pruningnumber of parameters having an absolute value of a parameter value notgreater than the parameter value threshold for pruning in ascendingorder of the absolute value of the parameter value, that is, setting theselected parameter to the specified value. Finally, the electronicdevice may train the pruned neural network based on a present trainingsample using the machine learning method, so that the accuracy of thecurrent trained neural network is restored to the original accuracy asmuch as possible.

Here, when the electronic device is training the pruned neural network,at least one round of training operations may be performed. After eachround of training operations, the trained neural network may be used toperform a prediction operation on a preset test sample to determine theaccuracy of the neural network.

It should be noted that the electronic device may fine-tune the neuralnetwork when training the pruned neural network. The advantage offine-tuning is that the training efficiency may be improved withoutcompletely retraining the neural network, and a better result may beobtained after a relatively small number of iterations. For example,with a reduced number of parameters of the neural network, the currentaccuracy is close to the original accuracy.

In some alternative implementations of the present embodiment, for eachto-be-compressed layer, the electronic device may select a parameter forpruning from the parameters included in the to-be-compressed layer basedon the pruning ratio and the parameter value threshold corresponding tothe to-be-compressed layer, and train the pruned neural network based onthe preset training sample using the machine learning method by thefollowing method:

First, the electronic device may perform the following pruning trainingoperations: determining the product of the specified ratio and the totalnumber of original parameters included in the to-be-compressed layer asthe pruning number, and selecting, from the parameters included in theto-be-compressed layer, the pruning number of parameters having anabsolute value of a parameter value not greater than the parameter valuethreshold for pruning in ascending order of the absolute value of theparameter value, wherein the specified ratio is not greater than thepruning ratio; training the pruned neural network based on the trainingsample using the machine learning method; and stopping execution of thepruning training operations in response to determining that the accuracyof the current trained neural network is lower than a preset accuracy,or the ratio of the number of the pruned parameters to the total numberin the to-be-compressed layer is not lower than the pruning ratio.

Here, the specified ratio is not greater than the pruning ratio. Theinitial value of the specified ratio may be manually set, or may be setby the electronic device, and the present embodiment does not have anylimitations with this respect. Here, if the pruning ratio is 70%, theinitial value of the specified ratio may be, for example, 40%, 20%, or10%. It should be noted that the preset accuracy may be the originalaccuracy of the neural network or a value slightly lower than theoriginal accuracy. The preset accuracy may be manually set, or may beset by the electronic device based on a preset algorithm. The presetaccuracy may be adjusted according to actual needs, and the presentembodiment does not have any limitations with this respect.

Then in response to determining that the accuracy of the current trainedneural network is not lower than the preset accuracy and the ratio isnot lower than the pruning ratio, the electronic device may reset thespecified ratio based on the pruning ratio and the ratio, and continuethe pruning training operations. For example, the electronic device maysubtract the ratio from the pruning ratio to obtain a first ratio, andreset the specified ratio to a positive number not greater than thefirst ratio. For example, assuming that the pruning ratio is 70% and theratio is 10%, then the first ratio may be 60%, and the electronic devicemay select a value in the interval (0, 60%] and reset the specifiedratio to this value.

In some alternative implementations of the present embodiment, theelectronic device may also reset the specified ratio to a positivenumber not greater than the first ratio and a current value of thespecified ratio. For example, if the current value of the specifiedratio is 10% and the first ratio is 60%, the electronic device mayselect a value (for example, 10%) in the interval (0, 60%] and reset thespecified ratio to this value.

In some alternative implementations of the present embodiment, theelectronic device may also reset the specified ratio to a positivenumber less than the current value of the specified ratio and notgreater than the first ratio. For example, if the current value of thespecified ratio is 30% and the first ratio is 40%, the electronic devicemay select a value (for example, 20) in the interval (0, 30%) and resetthe specified ratio to this value.

Step 204, determining the neural network obtained after performing theprocessing steps on the selected at least one to-be-compressed layer asa compressed neural network, and storing the compressed neural network.

In the present embodiment, the electronic device may determine theneural network obtained after performing the processing steps on theselected at least one to-be-compressed layer as a compressed neuralnetwork, and may store the compressed neural network, for example,storing locally on the electronic device (such as a hard disk or amemory) or on a server in remote communication connection to theelectronic device. Here, when storing the compressed neural network, thepruned parameter (that is, the parameter set as the specified value) isgenerally not required to be stored, and therefore, the storage spacemay be effectively saved.

With further reference to FIG. 3, FIG. 3 is a schematic diagram of anapplication scenario of the method for compressing a neural networkaccording to the present embodiment. In the application scenario of FIG.3, the parameter value threshold is 0.1. First, the server 301 maylocally acquire a to-be-compressed trained neural network 302. Here, theneural network 302 may be a neural network included in a to-be-releasedapplication suitable for mobile devices, and the neural network 302includes N layers, N is a natural number greater than zero. Then, theserver 301 may select each layer in the neural network 302 as theto-be-compressed layer, and obtain a set of to-be-compressed layers 303.The set of to-be-compressed layers 303 may include a to-be-compressedlayer 1, a to-be-compressed layer 2 . . . a to-be-compressed layer N.Then, the server 301 may perform the following processing stepssequentially on each of the to-be-compressed layers in the set ofto-be-compressed layers 303 in descending order of the number of levelof the to-be-compressed layer in the neural network 302: determining thepruning ratio based on the total number of parameters included in theto-be-compressed layer, selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and the parameter value threshold 0.1, and training the prunedneural network based on a preset training sample using the machinelearning method, so that the trained neural network may be restored tothe original accuracy as much as possible. Here, the selected parametersare parameters having the absolute value of parameter value not greaterthan 0.1, and the ratio of the number of selected parameters to thetotal number is not greater than the pruning ratio. Finally, the server301 may determine the neural network obtained after performing theprocessing steps on the selected at least one to-be-compressed layer inthe set of to-be-compressed layers 303 as a compressed neural network304, and store the compressed neural network 304 locally.

The method provided by the embodiments of the present disclosureeffectively utilizes the specified processing steps performed on theselected to-be-compressed layer to appropriately prune the parameters inthe to-be-compressed layer, and trains the current neural network basedon a preset training sample using the machine learning method after theparameters are pruned, so that the neural network may be restored to theoriginal accuracy as much as possible, thereby achieving effectivecompression of the neural network.

With further reference to FIG. 4, as an implementation to the methodshown in the above figures, the present disclosure provides anembodiment of an apparatus for compressing a neural network. Theapparatus embodiment corresponds to the method embodiment shown in FIG.2, and the apparatus may specifically be applied to various electronicdevices.

As shown in FIG. 4, the apparatus 400 for compressing a neural networkof the present embodiment includes: an acquisition unit 401, a selectionunit 402, a processing unit 403 and a storing unit 404. Here, theacquisition unit 401 is configured to acquire a to-be-compressed trainedneural network. The selection unit 402 is configured to select at leastone layer from layers of the neural network as a to-be-compressed layer.The processing unit 403 is configured to perform the followingprocessing steps sequentially on each of the to-be-compressed layers indescending order of the number of level of the to-be-compressed layer inthe neural network: determining a pruning ratio based on a total numberof parameters included in the to-be-compressed layer, selecting aparameter for pruning from the parameters included in theto-be-compressed layer based on the pruning ratio and a parameter valuethreshold, and training the pruned neural network based on a presettraining sample using a machine learning method. The storing unit 404 isconfigured to determine the neural network obtained after performing theprocessing steps on the selected at least one to-be-compressed layer asa compressed neural network, and store the compressed neural network.

In the present embodiment, in the apparatus 400 for compressing a neuralnetwork: the specific processing and the technical effects thereof ofthe acquisition unit 401, the selection unit 402, the processing unit403, and the storing unit 404 may be referred to the relateddescriptions of step 201, step 202, step 203, and step 204 in thecorresponding embodiment of FIG. 2, respectively, and detaileddescription thereof will be omitted.

In some alternative implementations of the present embodiment, theselection unit 402 may include: a selection subunit (not shown in thefigure), configured to select, in response to the neural networkincluding a convolutional layer and a fully connected layers, at leastone of at least one convolutional layer or at least one fully connectedlayers as the to-be-compressed layer.

In some alternative implementations of the present embodiment, theprocessing unit 403 may include: a first processing subunit (not shownin the figure), configured to perform the following pruning trainingoperations: determining a product of a specified ratio and the totalnumber as a pruning number, and selecting, from the parameters includedin the to-be-compressed layer, the pruning number of parameters havingan absolute value of a parameter value not greater than the parametervalue threshold for pruning in ascending order of the absolute value ofthe parameter value, wherein the specified ratio is not greater than thepruning ratio; training the pruned neural network based on the trainingsample using the machine learning method; and stopping execution of thepruning training operations in response to determining that an accuracyof the current trained neural network is lower than a preset accuracy,or the ratio of the number of the pruned parameters to the total numberin the to-be-compressed layer is not lower than the pruning ratio; and asecond processing subunit (not shown in the figure), configured to resetthe specified ratio based on the pruning ratio and the ratio, andcontinue the pruning training operations in response to determining thatthe accuracy of the current trained neural network is not lower than thepreset accuracy and the ratio is lower than the pruning ratio.

In some alternative implementations of the present embodiment, thesecond processing subunit may include: a setting module (not shown inthe figure), configured to subtract the ratio from the pruning ratio toobtain a first ratio, and reset the specified ratio to a positive numbernot greater than the first ratio.

In some alternative implementations of the present embodiment, thesetting module may include: a setting submodule not shown in thefigure), configured to reset the specified ratio to a positive numbernot greater than the first ratio and a current value of the specifiedratio.

In some alternative implementations of the present embodiment, thesetting submodule may be further configured to: reset the specifiedratio to a positive number less than the current value of the specifiedratio and not greater than the first ratio.

The apparatus provided by the embodiments of the present disclosureeffectively utilizes the specified processing steps performed on theselected to-be-compressed layer to appropriately prune the parameters inthe to-be-compressed layer, and trains the current neural network basedon a preset training sample using the machine learning method after theparameters are pruned, so that the neural network may be restored to theoriginal accuracy as much as possible, thereby achieving effectivecompression of the neural network.

Referring to FIG. 5, a schematic structural diagram of a computer system500 adapted to implement an electronic device of the embodiments of thepresent application is shown. The electronic device shown in FIG. 5 ismerely an example and should not impose any restriction on the functionand scope of use of the embodiments of the present application.

As shown in FIG. 5, the computer system 500 includes a centralprocessing unit (CPU) 501, which may execute various appropriate actionsand processes in accordance with a program stored in a read-only memory(ROM) 502 or a program loaded into a random access memory (RAM) 503 froma storage portion 508. The RAM 503 also stores various programs and datarequired by operations of the system 500. The CPU 501, the ROM 502 andthe RAM 503 are connected to each other through a bus 504. Aninput/output (I/O) interface 505 is also connected to the bus 504.

The following components are connected to the I/O interface 505: aninput portion 506 including a keyboard, a mouse etc.; an output portion507 comprising a cathode ray tube (CRT), a liquid crystal display device(LCD), a speaker etc.; a storage portion 508 including a hard disk andthe like; and a communication portion 509 comprising a network interfacecard, such as a LAN card and a modem. The communication portion 509performs communication processes via a network, such as the Internet. Adrive 510 is also connected to the I/O interface 505 as required. Aremovable medium 511, such as a magnetic disk, an optical disk, amagneto-optical disk, and a semiconductor memory, may be installed onthe drive 510, to facilitate the retrieval of a computer program fromthe removable medium 511, and the installation thereof on the storageportion 508 as needed.

In particular, according to embodiments of the present disclosure, theprocess described above with reference to the flow chart may beimplemented in a computer software program. For example, an embodimentof the present disclosure includes a computer program product, whichcomprises a computer program that is tangibly embedded in amachine-readable medium. The computer program comprises program codesfor executing the method as illustrated in the flow chart. In such anembodiment, the computer program may be downloaded and installed from anetwork via the communication portion 509, and/or may be installed fromthe removable media 511. The computer program, when executed by thecentral processing unit (CPU) 501, implements the above mentionedfunctionalities as defined by the methods of the present disclosure.

It should be noted that the computer readable medium in the presentdisclosure may be computer readable storage medium. An example of thecomputer readable storage medium may include, but not limited to:semiconductor systems, apparatus, elements, or a combination any of theabove. A more specific example of the computer readable storage mediummay include but is not limited to: electrical connection with one ormore wire, a portable computer disk, a hard disk, a random access memory(RAM), a read only memory (ROM), an erasable programmable read onlymemory (EPROM or flash memory), a fibre, a portable compact disk readonly memory (CD-ROM), an optical memory, a magnet memory or any suitablecombination of the above. In the present disclosure, the computerreadable storage medium may be any physical medium containing or storingprograms which can be used by a command execution system, apparatus orelement or incorporated thereto. The computer readable medium may be anycomputer readable medium except for the computer readable storagemedium. The computer readable medium is capable of transmitting,propagating or transferring programs for use by, or used in combinationwith, a command execution system, apparatus or element. The programcodes contained on the computer readable medium may be transmitted withany suitable medium including but not limited to: wireless, wired,optical cable, RF medium etc., or any suitable combination of the above.

The flow charts and block diagrams in the accompanying drawingsillustrate architectures, functions and operations that may beimplemented according to the systems, methods and computer programproducts of the various embodiments of the present disclosure. In thisregard, each of the blocks in the flow charts or block diagrams mayrepresent a module, a program segment, or a code portion, said module,program segment, or code portion comprising one or more executableinstructions for implementing specified logic functions. It should alsobe noted that, in some alternative implementations, the functionsdenoted by the blocks may occur in a sequence different from thesequences shown in the figures. For example, any two blocks presented insuccession may be executed, substantially in parallel, or they maysometimes be in a reverse sequence, depending on the function involved.It should also be noted that each block in the block diagrams and/orflow charts as well as a combination of blocks may be implemented usinga dedicated hardware-based system executing specified functions oroperations, or by a combination of a dedicated hardware and computerinstructions.

The units or modules involved in the embodiments of the presentapplication may be implemented by means of software or hardware. Thedescribed units or modules may also be provided in a processor, forexample, described as: a processor, comprising an acquisition unit, aselection unit, a processing unit, and a storing unit, where the namesof these units or modules do not in some cases constitute a limitationto such units or modules themselves. For example, the acquisition unitmay also be described as “a unit for acquiring a to-be-compressedtrained neural network.”

In another aspect, the present application further provides anon-transitory computer-readable storage medium. The non-transitorycomputer-readable storage medium may be the non-transitorycomputer-readable storage medium included in the apparatus in the abovedescribed embodiments, or a stand-alone non-transitory computer-readablestorage medium not assembled into the apparatus. The non-transitorycomputer-readable storage medium stores one or more programs. The one ormore programs, when executed by a device, cause the device to: acquire ato-be-compressed trained neural network; select at least, one layer fromlayers of the neural network as a to-be-compressed layer; performfollowing processing steps sequentially on each of the to-be-compressedlayers in descending order of a number of level of the to-be-compressedlayer in the neural network: determining a pruning ratio based on atotal number of parameters included in the to-be-compressed layer,selecting a parameter for pruning from the parameters included in theto-be-compressed layer based on the pruning ratio and a parameter valuethreshold, and training the pruned neural network based on a presettraining sample using a machine learning method; and determine theneural network obtained after performing the processing steps on theselected at least one to-be-compressed layer as a compressed neuralnetwork, and store the compressed neural network.

The above description only provides an explanation of the preferredembodiments of the present application and the technical principlesused. It should be appreciated by those skilled in the art that theinventive scope of the present application is not limited to thetechnical solutions formed by the particular combinations of theabove-described technical features. The inventive scope should alsocover other technical solutions formed by any combinations of theabove-described technical features or equivalent features thereofwithout departing from the concept of the disclosure. Technical schemesformed by the above-described features being interchanged with, but notlimited to, technical features with similar functions disclosed in thepresent application are examples.

What is claimed is:
 1. A method for compressing a neural network, themethod comprising: acquiring a to-be-compressed trained neural network;selecting at least one layer from layers of the neural network as ato-be-compressed layer; performing following processing stepssequentially on each of the to-be-compressed layers in descending orderof a number of level of the to-be-compressed layer in the neuralnetwork: determining a pruning ratio based on a total number ofparameters included in the to-be-compressed layer, selecting a parameterfor pruning from the parameters included in the to-be-compressed layerbased on the pruning ratio and a parameter value threshold, and trainingthe pruned neural network based on a preset training sample using amachine learning method; and determining the neural network obtainedafter performing the processing steps on the selected at least oneto-be-compressed layer as a compressed neural network, and storing thecompressed neural network; wherein selecting a parameter for pruningfrom the parameters included in the to-be-compressed layer based on thepruning ratio and a parameter value threshold, and training the prunedneural network based on a preset training sample using a machinelearning method comprises: performing following pruning trainingoperations: determining a product of a specified ratio and the totalnumber as a pruning number, and selecting, from the parameters includedin the to-be-compressed layer, the pruning number of parameters havingan absolute value of a parameter value not greater than the parametervalue threshold for pruning in ascending order of the absolute value ofthe parameter value, wherein the specified ratio is not greater than thepruning ratio; training the pruned neural network based on the trainingsample using the machine learning method; and subtracting the ratio fromthe pruning ratio to obtain a first ratio, resetting the specified ratioto a positive number not greater than the first ratio, and continuingthe pruning training operations in response to determining that theaccuracy of the current trained neural network is not lower than thepreset accuracy and the ratio is lower than the pruning ratio.
 2. Themethod according to claim 1, wherein the selecting at least one layerfrom layers of the neural network as a to-be-compressed layer comprises:selecting, in response to the neural network comprising a convolutionallayer and a fully connected layers, at least one of at least oneconvolutional layer or at least one fully connected layers as theto-be-compressed layer.
 3. The method according to claim 1, wherein theselecting a parameter for pruning from the parameters included in theto-be-compressed layer based on the pruning ratio and a parameter valuethreshold, and training the pruned neural network based on a presettraining sample using a machine learning method further comprises:stopping execution of the pruning training operations in response todetermining that an accuracy of the current trained neural network islower than a preset accuracy, or a ratio of a number of the prunedparameters to the total number in the to-be-compressed layer is notlower than the pruning ratio.
 4. The method according to claim 1,wherein the resetting the specified ratio to a positive number notgreater than the first ratio comprises: resetting the specified ratio toa positive number not greater than the first ratio and a current valueof the specified ratio.
 5. The method according to claim 4, wherein theresetting the specified ratio to a positive number not greater than thefirst ratio and a current value of the specified ratio comprises:resetting the specified ratio to a positive number less than the currentvalue and not greater than the first ratio.
 6. An apparatus forcompressing a neural network, the apparatus comprising: at least oneprocessor; and a memory storing instructions, the instructions whenexecuted by the at least one processor, cause the at least one processorto perform operations, the operations comprising: acquiring ato-be-compressed trained neural network; selecting at least one layerfrom layers of the neural network as a to-be-compressed layer;performing following processing steps sequentially on each of theto-be-compressed layers in descending order of a number of level of theto-be-compressed layer in the neural network: determining a pruningratio based on a total number of parameters included in theto-be-compressed layer, selecting a parameter for pruning from theparameters included in the to-be-compressed layer based on the pruningratio and a parameter value threshold, and training the pruned neuralnetwork based on a preset training sample using a machine learningmethod; and determining the neural network obtained after performing theprocessing steps on the selected at least one to-be-compressed layer asa compressed neural network, and storing the compressed neural network,wherein selecting a parameter for pruning from the parameters includedin the to-be-compressed layer based on the pruning ratio and a parametervalue threshold, and training the pruned neural network based on apreset training sample using a machine learning method comprises:performing following pruning training operations: determining a productof a specified ratio and the total number as a pruning number, andselecting, from the parameters included in the to-be-compressed layer,the pruning number of parameters having an absolute value of a parametervalue not greater than the parameter value threshold for pruning inascending order of the absolute value of the parameter value, whereinthe specified ratio is not greater than the pruning ratio; training thepruned neural network based on the training sample using the machinelearning method; and subtracting the ratio from the pruning ratio toobtain a first ratio, resetting the specified ratio to a positive numbernot greater than the first ratio, and continuing the pruning trainingoperations in response to determining that the accuracy of the currenttrained neural network is not lower than the preset accuracy and theratio is lower than the pruning ratio.
 7. The apparatus according toclaim 6, wherein the selecting at least one layer from layers of theneural network as a to-be-compressed layer comprises: selecting, inresponse to the neural network comprising a convolutional layer and afully connected layers, at least one of at least one convolutional layeror at least one fully connected layers as the to-be-compressed layer. 8.The apparatus according to claim 6, wherein the selecting a parameterfor pruning from the parameters included in the to-be-compressed layerbased on the pruning ratio and a parameter value threshold, and trainingthe pruned neural network based on a preset training sample using amachine learning method further comprises: stopping execution of thepruning training operations in response to determining that an accuracyof the current trained neural network is lower than a preset accuracy,or a ratio of a number of the pruned parameters to the total number inthe to-be-compressed layer is not lower than the pruning ratio.
 9. Theapparatus according to claim 6, wherein the resetting the specifiedratio to a positive number not greater than the first ratio comprises:resetting the specified ratio to a positive number not greater than thefirst ratio and a current value of the specified ratio.
 10. Theapparatus according to claim 9, wherein the resetting the specifiedratio to a positive number not greater than the first ratio and acurrent value of the specified ratio comprises: resetting the specifiedratio to a positive number less than the current value and not greaterthan the first ratio.
 11. A non-transitory computer-readable storagemedium storing a computer program, the computer program when executed byone or more processors, causes the one or more processors to performoperations, the operations comprising: acquiring a to-be-compressedtrained neural network; selecting at least one layer from layers of theneural network as a to-be-compressed layer; performing followingprocessing steps sequentially on each of the to-be-compressed layers indescending order of a number of level of the to-be-compressed layer inthe neural network: determining a pruning ratio based on a total numberof parameters included in the to-be-compressed layer, selecting aparameter for pruning from the parameters included in theto-be-compressed layer based on the pruning ratio and a parameter valuethreshold, and training the pruned neural network based on a presettraining sample using a machine learning method; and determining theneural network obtained after performing the processing steps on theselected at least one to-be-compressed layer as a compressed neuralnetwork, and storing the compressed neural network, wherein selecting aparameter for pruning from the parameters included in theto-be-compressed layer based on the pruning ratio and a parameter valuethreshold, and training the pruned neural network based on a presettraining sample using a machine learning method comprises: performingfollowing pruning training operations: determining a product of aspecified ratio and the total number as a pruning number, and selecting,from the parameters included in the to-be-compressed layer, the pruningnumber of parameters having an absolute value of a parameter value notgreater than the parameter value threshold for pruning in ascendingorder of the absolute value of the parameter value, wherein thespecified ratio is not greater than the pruning ratio; training thepruned neural network based on the training sample using the machinelearning method; and subtracting the ratio from the pruning ratio toobtain a first ratio, resetting the specified ratio to a positive numbernot greater than the first ratio, and continuing the pruning trainingoperations in response to determining that the accuracy of the currenttrained neural network is not lower than the preset accuracy and theratio is lower than the pruning ratio.